WIP: spike consolidating how we run Smokey test suite #1231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I plan to tag up all scenarios that should pass when run against the failover CDN (Cloudfront). These scenarios can then be tested in the Cloudfront 'context', by setting a `GOVUK_WEBSITE_ROOT` of "secret.cloudfront.net" (where "secret" is replaced by the actual distribution ID that can be found at govuk-dns-tf#69), provided all requests include a `Host` header of `www.gov.uk`. [Selenium doesn't support setting request headers](https://stackoverflow.com/a/15647143), and thus requires use of a proxy like browsermob-proxy, to inject headers. I _really_ don't want to use browsermob-proxy - it was already in Smokey a couple of years ago, but was [removed because of its flakiness and Java dependency](#924) There are some [HTTP request support methods in Smokey](https://github.com/alphagov/smokey/blob/2319cca9a815ff40ba5eed8483bfc4089a04a480/features/support/http_requests.rb), which bypass Capybara and have support for custom headers. I did [investigate reusing those methods](#1220) but it would have meant rewriting our entire test suite (to stop using Capybara). Therefore, I built a lightweight proxy, which we can set as the `GOVUK_WEBSITE_ROOT` and pass custom headers to as needed, without the need to rewrite our step definitions. In theory, we should now be able to delete quite a lot of the custom HTTP request support code, and make the tests use standard Capybara/Selenium methods (which could now use our proxy under the hood instead).
We have not reproduced Fastly's VCL behaviour for things like: - Cookie banner - A/B tests - Feedback/Smartsurvey So we should not run these particular Smokey tests against Cloudfront.
These scenarios pass when tested against the mirror host. Other scenarios will never pass on the mirror (e.g. scenarios which rely on search or logging in). I had to add a `sleep` to allow time for the meta refresh to happen. Difficult to do much else at the proxy.rb layer, as the response is a 200, so as far as the proxy is concerned there is no redirect to wait for, so it feels sensible to handle this at the step definition layer. However, we can utilise Capybara under the hood. The `.has_css?` method will wait until an element matching the selector is found. `govuk-template` is the class name on the `<html>` element for all GOV.UK HTML pages, so should be a reliable indicator that the browser has followed all 301 redirects.
Some assets are returning a 503 when going through the proxy to the
mirror. Presumably the assets are missing from the mirror, and it
is AWS/GCP that is returning the 503 as a result.
It's not important for functionality, so let's just stub it.
Fixing the assets issue is outside of the scope of this PR.
```
Scenario: Check the frontend can talk to Content Store # features/apps/finder_frontend.feature:7
When I visit "/government/people" # features/step_definitions/smokey_steps.rb:60
Then I should see "All ministers and senior officials on GOV.UK" # features/step_definitions/smokey_steps.rb:160
And I should see an input field to search # features/step_definitions/finder_frontend_steps.rb:1
Detected JS errors:
http://127.0.0.1:8080/assets/static/govuk_publishing_components/icon-close-eecb5fdf67d1688ddf2f93e6e09d5f87b9b24e0fc53596ad2a65cf187c2c7c76.svg - Failed to load resource: the server responded with a status of 503 (Service Unavailable) (RuntimeError)
./features/support/hooks.rb:31:in `After'
Failing Scenarios:
cucumber features/apps/finder_frontend.feature:7 # Scenario: Check the frontend can talk to Content Store
```
This is replaced by the newly tagged up scenarios that can be run against the mirrors (a much more comprehensive set of tests). It also means that the `if @responses` conditional branch of the `Then I should see` step definition is no longer used, and so has been deleted here.
These should theoretically work on the Cloudfront CDN. However, we have not made any edits to the `do_http_request` method to override the host and host headers (as we do in proxy.rb), so the tests fail. To avoid complicating an already non-standard Cucumber test, let's mark this as "not suitable for Cloudfront" for now, and consider how to refactor our tests further down the line, so that these two scenarios pass in both the primary and secondary CDN contexts.
Currently, we run Smokey through a proxy _only when testing against the failover CDN or mirrors_. The 'normal' Smokey tests that run against the primary CDN do not make use of the proxy. It's not ideal having two ways of running the tests. This commit is a proof of concept for getting ALL tests running through the proxy. Usage: ``` shell1$ env ENVIRONMENT="production" PROXY_PROFILE="primaryCDN" ruby proxy.rb shell2$ env ENVIRONMENT=production SIGNON_EMAIL="[email protected]" SIGNON_PASSWORD="replaceme" bundle exec cucumber --tags="not @notreplatforming" ``` The result is pretty good, with just a few quirky tests failing in production: ``` Failing Scenarios: cucumber features/apps/static.feature:4 # Scenario: Check the feedback component loads cucumber features/cdn.feature:3 # Scenario: Check all A/B test variants work cucumber features/cdn.feature:9 # Scenario: Check an A/B test is persistent cucumber features/cdn.feature:19 # Scenario: Check caching behaviour for POST requests cucumber features/cdn.feature:25 # Scenario: Check caching behaviour for GET requests ``` The issues are more fundamental for Staging and Integration, the former needing to be accessed via VPN and the latter needing to satisfy Basic Authentication. *Needs further work*.
ChrisBAshton
force-pushed
the
cdn-failover-test-proxy
branch
from
7962e4f
to
2976dc5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, we run Smokey through a proxy only when testing against the failover CDN or mirrors. The 'normal' Smokey tests that run against the primary CDN do not make use of the proxy. It's not ideal having two ways of running the tests.
This commit is a proof of concept for getting ALL tests running through the proxy. Usage:
The result is pretty good, with just a few quirky tests failing in production:
The issues are more fundamental for Staging and Integration, the former needing to be accessed via VPN and the latter needing to satisfy Basic Authentication. Needs further work.